Telephone versus Face-to-face Interviewing of National Probability Samples with Long Questionnaires Comparisons of Respondent Satisficing and Social Desirability Response Bias

The last 50 years have seen a gradual replacement of face-to-face interviewing with telephone interviewing as the dominant mode of survey data collection in the United States. But some of the most expensive and large-scale nationally funded, long-term survey research projects involving national area-probability samples and long questionnaires retain face-to-face interviewing as their mode. In this article, we propose two ways in which shifting such surveys to random digit dialing (RDD) telephone interviewing might affect the quality of data acquired, and we test these hypotheses using data from three national mode experiments. Random digit dialing telephone respondents were more likely to satisfice (as evidenced by no-opinion responding, nondifferentiation, and acquiescence), to be less cooperative and engaged in the interview, and were more likely to express dissatisfaction with the length of the interview than were face-to-face respondents, despite the fact that the telephone interviews were completed more quickly than the face-to-face interviews. Telephone respondents were also more suspicious about the interview process and more likely to present themselves in socially desirable ways than were face-to-face respondents. These findings shed light on the nature of the survey response process, on the costs and benefits associated with particular survey modes, and on the nature of social interaction generally. allyson l. holbrook is Assistant Professor of Public Administration and Psychology at the Survey Research Laboratory at the University of Illinois at Chicago, melanie c. green is Assistant Professor of Psychology at the University of Pennsylvania, and jon a. krosnick is Professor of Psychology and Political Science at The Ohio State University and University Fellow at Resources for the Future. The authors thank Nancy Burns, Donald Kinder, Virginia Sapiro, 80 Holbrook, Green, and Krosnick Fifty years ago, the vast majority of high-quality surveys in America were conducted via face-to-face interviewing. But following the widespread introduction of the telephone in the United States in the mid-twentieth century, survey researchers began a swift shift to conducting surveys via telephone instead. Today, most local, regional, national, and listed sample surveys are conducted by telephone. When researchers conducting such surveys seek representative general population samples, they most often sample via random digit dialing (RDD). The appeal of telephone interviewing is multifaceted, because this method has many practical advantages, most notably reduced cost, the possibility of quick turnaround time, and the possibility of closer supervision of interviewers to assure greater standardization of administration. Initially, telephone interviewing had another unique advantage as well: the possibility of computerdriven questionnaire presentation. With the advent of Computer Assisted Personal Interviewing (CAPI), telephone interviewing’s edge in this regard is gone, but this mode continues to maintain its other unique advantages and its popularity in practice. Telephone interviewing has obvious disadvantages, too. For example, show cards, which are often used to present response choices in face-to-face interviews, are more difficult to employ in telephone surveys, requiring advance contact, mailing of cards to respondents, and respondent responsibility for manipulating the cards during the interview. Therefore, telephone surveys routinely forgo the use of show cards (but see Miller [1984] for a discussion of the effects of this omission and Groves and Kahn [1979] for a discussion of possible disadvantages of show cards). As of 1998, about 5 percent of the U.S. population did not have a working telephone in their household, thereby prohibiting these individuals from participating in telephone surveys (Belinfante 1998). And for a variety of reasons, it has always been more difficult to obtain response rates in telephone surveys as high as those obtained in face-to-face surveys (e.g., Groves 1977; Mulry-Liggan 1983; Shanks, Sanchez, and Morton 1983; Weeks et al. 1983). Thus, it is not obvious that data quality in telephone surveys will meet or exceed that obtained from face-toface surveys. Kathy Cirksena, James Lepkowski, Robert Belli, Robert Groves, Robert Kahn, John Van Hoyke, Ashley Grosse, Charles Ellis, Paul Biemer, the members of the National Election Study Ad Hoc Committee on Survey Mode (Norman Bradburn, Charles Franklin, Graham Kalton, Merrill Shanks, and Sidney Verba), and the members of the National Election Study Board of Overseers for their help, encouragement, and advice. We are also grateful to Aldena Rogers and Chris Mesmer for their assistance in collecting data for one of the social desirability validation studies. This research was supported by a grant from the National Science Foundation (SBR-9707741). Correspondence should be addressed to Allyson L. Holbrook at Survey Research Laboratory, 412 S. Peoria St., Sixth Floor, Chicago, IL 60607-7069 (e-mail: [email protected]), Jon A. Krosnick at the Department of Psychology, Ohio State University, 1885 Neil Avenue, Columbus, OH 43210 (e-mail: [email protected]), or Melanie Green at the Department of Psychology, University of Pennsylvania, 3815 Walnut Street, Philadelphia, PA 19104-6196 (e-mail: [email protected]). Mode Effects on Satisficing and Social Desirability 81 Perhaps for such reasons, a diverse group of the nation’s most expensive, long-term, large-scale, federally funded survey studies of national samples involving long questionnaires have retained the face-to-face method while most other survey research moved to the telephone. For example, the National Election Studies have conducted face-to-face interviews since the 1940s; the General Social Survey has done so since 1972. The National Health Interview Survey (conducted by the U.S. Census Bureau for the Centers for Disease Control and Prevention), the National Crime Victimization Survey (conducted by the U.S. Census Bureau for the Bureau of Justice Statistics), and many other such large survey projects sponsored by government agencies have done so as well. In this article, we explore whether there are potential benefits of such continued reliance on face-to-face interviewing of national area probability samples for studies with long questionnaires, as compared to the obvious alternative of RDD telephone interviewing. We focus in particular on two possible sources of response error differences. First, we consider the possibility that face-to-face respondents may be more likely to exert the required cognitive effort to answer questions carefully, whereas telephone respondents may be less likely to do so. As a result, the latter individuals may manifest more survey satisficing, thereby compromising response quality. Second, we consider the possibility that face-to-face respondents may differ from telephone respondents in the comfort they have in reporting socially undesirable attitudes, beliefs, or behaviors. As a result, the magnitude of social desirability response bias may differ between the modes. We begin below by outlining the theoretical rationales underlying these hypotheses. Then, we review the results of many past studies that at first seem to offer evidence regarding these hypotheses. But as we will explain, the designs of these studies render most of them uninformative about the issues of interest here. We therefore proceed to describe the results of new tests of the satisficing and social desirability hypotheses using data from three largescale experiments that involved long interviews of representative national samples.